Skip to content

fix: make Noema review independent - #1477

Merged
seonghobae merged 3 commits into
mainfrom
fix/noema-independent-review
Aug 31, 2026
Merged

fix: make Noema review independent#1477
seonghobae merged 3 commits into
mainfrom
fix/noema-independent-review

Conversation

@seonghobae

@seonghobae seonghobae commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Summary

  • run Noema without waiting for an OpenCode approval, review threads, or other checks
  • fail closed on a colliding reviewer credential or empty LLM review content
  • pin the shared review sidecar to contextual-orchestrator merge 8cd99f139915131ba0239bce12a5d6a5fd85394e
  • amend ADR 0003 with the independent-review contract

Evidence

  • uv run python -m pytest -q: 2094 passed, 1 skipped, 21 subtests passed
  • focused Noema/sidecar contracts: 47 passed
  • independent agent review found and prompted closure of the bare-decision green path

Review semantics

Provider families are not introduced. Provider accounts remain independent credentials; model family semantics belong only to model_group.

No research PDF is added: this is a CI correctness and governance repair grounded in repository runtime evidence, not a substantive routing algorithm change.


Devin Review

Signed-off-by: Seongho Bae <me@seonghobae.me>
@coderabbitai

coderabbitai Bot commented Aug 31, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 6 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 0302b57a-4a9e-4223-9e1a-3d739c45ef4e

📥 Commits

Reviewing files that changed from the base of the PR and between 4b115bd and a80b318.

📒 Files selected for processing (9)
  • .github/workflows/noema-review.yml
  • docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md
  • scripts/ci/contextual_orchestrator_review_sidecar.sh
  • scripts/ci/noema_review_gate.py
  • tests/test_contextual_orchestrator_review_sidecar_contract.py
  • tests/test_noema_review_gate.py
  • tests/test_repository_branch_coverage_javascript_and_noema.py
  • tests/test_repository_branch_coverage_reporting_edges.py
  • tests/test_required_workflow_queue_contract.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 5 potential issues.

Devin Review

Comment thread scripts/ci/noema_review_gate.py
Comment thread scripts/ci/noema_review_gate.py
Comment thread scripts/ci/noema_review_gate.py
Comment thread scripts/ci/noema_review_gate.py
set -euo pipefail

ORCHESTRATOR_PIN_SHA="${ORCHESTRATOR_PIN_SHA:-c107e3e52371993aa9c326fcc245e01c41fc3850}"
ORCHESTRATOR_PIN_SHA="${ORCHESTRATOR_PIN_SHA:-8cd99f139915131ba0239bce12a5d6a5fd85394e}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 External pin compatibility unverified

The runtime, ADR, and test pins agree. The vendored commit is absent locally, so its API and lock compatibility still depend on external integration evidence.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified: the pinned commit is the merged contextual-orchestrator #949 SHA 8cd99f139915131ba0239bce12a5d6a5fd85394e. Its exact-head full suite (2830 passed, 1 skipped), Atheris, supply-chain, CodeQL, Semgrep, Trivy and OSV checks passed before merge; this PR's sidecar contracts and full central suite also pass.

Signed-off-by: Seongho Bae <me@seonghobae.me>

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 2 new potential issues.

Devin Review

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Obsolete check data remains queried

PR_QUERY still fetches reviewDecision and the status-check rollup after their consumers were removed. Drop them to match the independent-review boundary and reduce GraphQL cost.

(Refers to this code)

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +23 to 27
github.event.pull_request.base.repo.full_name ||
github.event.client_payload.target_repository || github.repository }}-${{
github.event.pull_request.number || github.event.workflow_run.pull_requests[0].number ||
github.event.client_payload.pr_number ||
github.run_id }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Follow-ups cancel required Noema checks

When OpenCode or Strix finishes during a pull-request Noema run, the shared concurrency.group cancels that required run. Its workflow-run replacement cannot restore the cancelled pull-request check.

Prompt for agents
Keep pull_request_target Noema runs in a concurrency group that workflow_run and repository_dispatch follow-ups cannot cancel, because only the pull-request-triggered run supplies the required PR check. Follow-up triggers can still be deduplicated separately, and the close-event run must retain a way to cancel the active pull-request run. Update the queue contract tests to assert both required-check isolation and close-event cancellation.
Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Signed-off-by: Seongho Bae <me@seonghobae.me>
@seonghobae

Copy link
Copy Markdown
Contributor Author

Current-head follow-up a80b3182:

  • Removed obsolete OpenCode-gating policy helpers.
  • Serialized all Noema trigger types by repository/PR.
  • Failed closed on unknown reviewer identity.
  • Strictly validated finding severity, file, line, and message; request_changes requires at least one substantive finding.
  • CI-equivalent local gate: 2105 passed, 1 skipped, 21 subtests passed; 10390 statements and 4138 branches at 100% coverage.

Noema remains independent of OpenCode approval and other check state.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 0 new potential issues.

Devin Review

@seonghobae
seonghobae merged commit a643d8a into main Aug 31, 2026
51 of 52 checks passed
@seonghobae
seonghobae deleted the fix/noema-independent-review branch August 31, 2026 05:33
seonghobae pushed a commit that referenced this pull request Aug 31, 2026
… cap no-op bug

Rebases this branch onto main's provider_account/account_cap rename (#1468)
and Noema-independence work (#1477/#1480), then fixes a real bug this
branch's own batched-preflight merge introduced: _catalog_family_cap()
defaulted to REVIEW_PREFLIGHT_MAX_TOTAL_ROUTES (24) whenever
ORCHESTRATOR_CATALOG_FAMILY_CAP was unset, silently disabling per-account
diversification. Live production evidence: probed_count 12, ready_count 2,
rejected_count 10 (83% rejected via 429/404/timeout), with the admitted
free-pool catalog 100% nvidia_nim/nvidia_nim_sub -- two credentials sharing
one rate-limited upstream jointly occupying the entire 12-slot preflight
batch. Reported at
#1415 (comment)

Fix (mirrors the pattern in open main PR #1487's diff, not yet merged):
renamed the helper to _catalog_account_cap(default), which now requires the
caller to supply contextual_orchestrator_review_policy.DEFAULT_ACCOUNT_CAP
(4) as the default instead of hand-typing/defaulting to a total-routes-scale
constant. An explicit ORCHESTRATOR_CATALOG_ACCOUNT_CAP override remains
honored. The sidecar shell script's equivalent CATALOG_FAMILY_CAP default
(also baked to the total 24-route budget) is renamed to CATALOG_ACCOUNT_CAP
and restored to a real, smaller cap (8, main's existing operational value).

Also corrects CHANGELOG.md and docs/adr/0003 prose that had described the
no-op 24-route cap as an intentional fix, and updates two contract tests
(test_contextual_orchestrator_review_runtime_preflight.py,
test_contextual_orchestrator_review_sidecar_contract.py) that pinned the old
family_cap naming and the buggy default.

Verification: coverage run -m pytest tests -> 2122 passed, 1 skipped, 21
subtests; coverage report -> 100% on scripts/ci; interrogate -> 100%
docstrings; bash -n on the touched shell script; git diff --check clean.
Red-then-green: reverting _catalog_account_cap's default back to
REVIEW_PREFLIGHT_MAX_TOTAL_ROUTES makes the new regression tests fail;
restoring the fix makes them pass again.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX
seonghobae pushed a commit that referenced this pull request Sep 2, 2026
Devin Review (this PR, scripts/ci/contextual_orchestrator_review_launcher.py
lines 129-169) flagged that the docstrings describe
#1477 (the ORCHESTRATOR_PIN_SHA bump to
contextual-orchestrator#949's merge commit) as still pending. Verified: #1477
merged on 2026-08-31 -- before this PR's original evidence_only/spend_admitted
fix even landed -- and `git merge-base --is-ancestor
8cd99f139915131ba0239bce12a5d6a5fd85394e 045d17da5e2aea56a97e241ee158ab1628d78660`
against contextual-orchestrator confirms the current pin already descends from
#949's fix commit. The "not yet pinned" / "once #1477 merges" framing was
therefore stale runtime history that would mislead future maintenance into
thinking the pin bump was still outstanding.

Updates both affected docstrings in the launcher module and the two
docstrings in tests/test_contextual_orchestrator_review_runtime_preflight.py
that repeated the same stale framing. Docs-only: no behavior change, no
assertion changed. docs/product-technical-gap-baseline.md's own #1477
references are left untouched -- they are dated, explicitly time-qualified
("open as of this correction") historical entries in this repo's append-only
decision log, not living claims about current state, so rewriting them would
falsify the audit trail rather than correct it.

Verified: `python -m pytest tests/test_contextual_orchestrator_review_runtime_preflight.py
tests/test_contextual_orchestrator_review_sidecar_contract.py -q` -- 95 passed;
`interrogate scripts/ci/contextual_orchestrator_review_launcher.py` -- 100%.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Gs7KmNvH75nxz1sL8mKjw
seonghobae added a commit that referenced this pull request Sep 5, 2026
Two conflicted files, four blocks. The branch's delta splits into two
independent parts with opposite verdicts.

PART A - paginated review evidence: KEPT (49 of the branch's 52 added
script lines).
  main still fetches reviews through GraphQL 'reviews(last: 100)'
  (noema_review_gate.py:324), so a review event older than the last 100
  is invisible to existing_noema_review() - main's only consumer of
  pr['reviews']['nodes'] - and Noema re-reviews a PR it already reviewed.
  fetch_complete_reviews() and its fetch_pr wiring merged cleanly outside
  every conflict block and are unchanged here.

PART B - fail-closed guard before Noema skip paths: DROPPED as superseded
(3 of 52 lines).
  a643d8a 'fix: make Noema review independent (#1477)' deleted all four
  helpers this guard is built on. Verified org-wide, not just in this
  file: current_primary_approval, has_current_changes_requested,
  has_unresolved_threads and blocking_checks each return 0 hits across
  all of origin/main (positive control: require_expected_head, 16 hits).
  main closed the same hole by removing the skip paths rather than
  guarding them, and made the primary-actor case raise RuntimeError where
  the branch only returned 1 - strictly stronger than this PR's fix.
  The branch's own test also called the pre-#1477 signature
  (inspect_and_review missing required 'expected_head'), so it was
  superseded twice over.

Block verdicts:
- noema_review_gate.py x2  -> main's inspect_and_review structure. The
  branch's guard has no surviving call target.
- test file, block 1       -> main's test_existing_noema_review_matches_
  actor_and_head, plus the branch's two Part A tests appended at EOF (the
  conflict boundary cut through the middle of main's function, so they
  cannot be inserted at the block site).
- test file, block 2       -> main's parametrized cases; the branch's used
  make_pr shapes for deleted behaviour.
- test_noema_never_skips_success_without_current_head_primary_approval
  merged cleanly outside the blocks and was removed: it is Part B.

One assertion was translated, not dropped:
test_fetch_pr_keeps_exact_head_approval_older_than_one_hundred_reviews
asserted through the deleted current_primary_approval(). It now asserts
main's live path directly - len(pr['reviews']['nodes']) == 101 and
nodes[0] is the pre-page-boundary approval - which is the test's stated
purpose and keeps its detection power.

Evidence:
- uvx ruff check --select F821 scripts/ci tests: All checks passed
- full suite, branch head b8f4adb (unmerged): 2099 passed, 0 failed
- full suite, this merge:                      2899 passed, 0 failed
  (failure-name diff empty both ways, denominators 2099 and 2899)
- negative control: deleting the fetch_pr REST wiring makes the kept
  pagination test fail by name -> it detects the regression it claims to
- coverage: TOTAL 100%, noema_review_gate.py 860/860 100%
- interrogate: PASSED (minimum 100.0%)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
seonghobae added a commit that referenced this pull request Sep 5, 2026
…-20260824)

13 conflict blocks across four files. Both sides hardened the same function
independently - this branch at the credential egress boundary, main at output
validation and gateway telemetry (#1477) - so the resolution is a union, not a
choice, except where the two guards overlap.

THE OVERLAP, resolved in the branch's favour with evidence:
  main's reject_private_llm_url has a fail-OPEN path - 'except socket.gaierror:
  return' - and does not require HTTPS for non-loopback targets, does not reject
  userinfo, and denylists special addresses instead of requiring globally
  routable unicast. The branch's validate_endpoint is stricter on every one of
  those axes and fails closed on DNS failure, which is this PR's stated subject.
  So call_llm now calls validate_endpoint only; reject_private_llm_url stays
  defined because two tests exercise it directly and still pass.
  That makes main's four call_llm-level 'URL cannot target internal IP
  addresses' assertions wrong and the branch's replacements right - which is
  exactly what the branch's side of those four conflict blocks already said.

PORTED onto main's structure:
- opener gains ProxyHandler({}) + PinnedHTTPHandler/PinnedHTTPSHandler in front
  of main's NoRedirectHandler
- response.read(MAX_LLM_RESPONSE_BYTES + 1) with the over-limit check
- post-request DNS identity re-check before decoding
- constants and the pinned-connection classes merged alongside main's JSON
  bound helpers (disjoint symbol sets, so a plain union)

NOT PORTED, on policy:
  The branch also passes timeout=120 to opener.open and its oracle asserts
  'timeout == 120'. CLAUDE.md states model-path timeouts are policy-fixed, and
  #1889/#1890/#1892 each added such a cap and were all reverted (#1891, #1895);
  main's call_llm docstring says it 'carries no fixed model wall-clock
  deadline'. The assertion was translated to 'timeout is None' with that
  citation in a comment. The byte cap is the bound that test really needs.

TEST FIXTURES updated because the behaviour changed, not to make red go green:
- the branch's oracle called the pre-#1477 five-argument call_llm (11 sites) and
  did not stub validate_substantive_verdict, which main added; both translated,
  the latter the same way main's own call_llm tests do it
- fail-closed DNS means an unresolvable *.example.test aborts a test before its
  own assertion, so tests/test_noema_review_gate.py and
  tests/test_noema_repair_attempt_telemetry.py get an autouse fixture that
  resolves non-literal hosts and leaves literal addresses to the real resolver,
  so the internal-address tests keep their meaning
- the byte-capped read passes an argument, so nine response doubles became
  read(self, _limit=None)

Evidence:
- uvx ruff check --select F821 scripts/ci tests: All checks passed
- full suite, branch head 3feb583 (unmerged): 1931 passed, 0 failed
- full suite, this merge:                      2938 passed, 0 failed
- the branch's own oracle tests/test_noema_endpoint_boundary.py: 33 passed
  (10 before the signature translation)
- main's noema tests: 121 passed; telemetry contracts: 23 passed
- negative controls: removing the post-request DNS re-check fails
  test_public_endpoint_requires_https_and_stable_global_dns; removing the byte
  cap fails test_response_body_is_bounded_before_json_decoding - both by name
- coverage: TOTAL 100%; interrogate: PASSED (minimum 100.0%)

GAP FOUND, left for the author: removing PinnedHTTPHandler/PinnedHTTPSHandler
from the opener changes no test result (33 still pass). The pinned classes are
covered only by direct unit tests; nothing asserts they are installed in
call_llm's opener, so a future edit could unwire DNS pinning silently. This is
pre-existing in the PR, not introduced here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant